Mining : Foundations , Techniques and ApplicationsFinite - State Transducers for Semi - Structured Text

Text mining for semi-structured documents requires information extractors. Programming extractors by hand is diicult to catch up with the amount and the variation of the documents placed on the WorldWide Web everyday. This paper presents our recent result on applying machine learning techniques to au-tomatize the generation of the extractors. Our goal is to develop a domain and language independent approach that automatically learns an extractor from training examples of extraction. In particular, this paper discusses the use of nite-state transducers (FST) as the representation formalism of the extractors. Previously, we have shown that only a small number of examples is enough for learning perfect FST-based extractors for documents from a variety of real Web sites. In this paper, we introduce two classes of FST-based extractors: single-pass and multi-pass, and informally analyze their relative advantages and disadvantages. In terms of their sample complexity, we hypothesized that single-pass extractors may have an edge for tabular documents while multi-pass ex-tractors may require fewer examples for tagged-list documents. We veriied this hypothesis empirically and found that by slightly changing the problem setting, a single-pass learner can outperform a multi-pass learner for tagged-list documents. A prototype system based on this work will be presented in the demo session of this workshop.